Robust sound localization using multi-source audiovisual information fusion

This paper illustrates the synergic advantages of a multi-modal sound localization system utilizing 2 cameras and a 3-element microphone array. The 2 cameras were used as part of a stereo feature-detection based visual object localization system, while the microphones were combined to produce a sound localization system incorporating a Temporal Power Fusion (TPF) algorithm. The cameras and microphones were integrated using spatial likelihood functions (SLFs), which greatly simplifies the integration process. Test results show a significant improvement in the integrated vision and sound localization (IVSL) system’s ability over that of the stand-alone microphone-array based sound localization system to accurately localize sound sourcess in low signal to noise situations. The IVSL system maintained an average error of 15cm at signal-to-noise ratios as low as 0.5 dB.